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ABSTRACT 

The National Center for Education Statistics (NCES) 
could provide better information for policy makers by (1) analyzing 
likely uses of the information it collects and disseminates; (2) 
launching more detailed investigations of educational inputs and 
processes; (3) taking some steps to place findings on educational 
outcomes in context; and (4) continuing to attack the problem of 
quality control. To study the needs of information users, NCES could 
classify the concerns expressed by various constituencies, analyze 
the published indicators of educational quality, and draw analogies 
to statistical indicators in other fields such as economics. To more 
fully describe educational inputs and processes, two types of data 
are important: the simplest demographic statistics, as well as 
detailed data on the use of educational resources. To measure and 
report educational outcomes, the tests used should be described, 
outcomes such as dropout rate and the number of general equivalency 
degrees earned should be reported, and higher-order cognitive skills 
should be emphasized. To maintain quality control, NCES should 
prov.de quality data from its national surveys, and take a firm stand 
on the quality and consistency of data received from state education 
agencies. (GDC) 
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COMMENTS ON AN ELEMENTARY AND SECONOARY EDUCATION DATA PROGRAM 

Brenda J. Turnbull 
Policy Studies Associates 



The National Center for Education Statistics deserves credit 
for undertaking a fundamental redesign of its data program 
in elementary and secondary education. In this paper, I suggest 
that NCES use this opportunity to (1) analyze likely uses of the 
information it collects and disseminates, <2) launch more 
detailed investigations of educational inputs and processes, (3) 
take some steps to place findings on educational outcomes in 
context, and <4) continue to attack the problem of quality 
control • 

First, a word about my vantage point is in order. I am a 
researcher who collects and analyzes information about state 
and local implementation of programs for special-needs students 
and for educational quality. I try to draw from this information 
the types of conclusions that can help policymakers assess and 
modify programs. Thus I have a bias toward the information that 
policy audiences will use. I also like to see detailed 
information about what goes on in schools, which is not always 
conveyed by gross measures of the resources put into schools. 
Finally, because my familiarity with NCES is only that of an 
occasional user of its data, this paper may recommend some 
procedures that are already standard practice at the agency. If 
so, please consider those comments endorsements rather than 
recommendations • 



Addressing the Needs of Information Users 

Ideally, the collection of data should be driven by a 
framework of questions that the data will be used to answer. 
Working backwards from intended uses through projected analyses 
to the specification of data elements and methods of collecting 
them would result in an efficient and practical program. In 
reality, NCES has to live with uncertainty about the questions 
that will be posed. However, there are~se»e ways of reducing 
this uncertainty, in addition to soliciting advice through 
commissioned papers and hearings. Analysis of information 
needs could include the following approaches: 

o Identifying and classifying the concerns about 

elementary and secondary education recently expressed 
by important groups such as Congress, governors, state 
legislators, and the public. 

o Doing a content analysis of recent leports and reform 
proposals, looking for both the indicators that have 
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convinced their author* of the poor health of the 
education enterprise and the types of improvements in 
process and outcomes that they believe their 
recommendations could produce. 

o Drawing analogies in education to the statistics and 
indexes that are used in other fields, such as the 
gross national product, measures of housing starts, 
and the like. 

Classifying the conce rns of various constituencies 

One guide to the future interests of policymakers and the 
public is what they have said about education in the recent t<aet, 
since these groups' concerns remain relatively stable over time. 
The Gallup poll consistently show* that the public is concerned 
about discipline in their local schools, for example. The 
members of Congress for whom the educational opportunities of 
special-needs pupils have been an important concern are not 
abandoning their interest in these pupils. Governors and state 
legislatures will remain concerned that their states offer 
employers a skilled workforce. 

All these concerns can be translated into indicators that 
are worth collecting on a national scale. Citizens' worries 
about discipline suggest that it would be useful to have data on 
the frequency of various types of incidents in schools over 
time. Because the education of special-needs groups remains an 
issue in federal policy, data on educational resources, 
processes, and outcomes should be broken down by type of pupils 
wherever possible. The skills of entry-level workers, besides 
being a state concern, deserve some analysis on a national scale. 

Even some concerns that are primarily local can provide 
clues to data that are worth collecting nationally. Local 
administrators routinely collect and often use data on student 
attendance, the popularity of particular course offerings, and 
turnover among teachers and aides. Collecting some sort of data 
on matters such as these might be feasible and worthwhile on a 
national scale since these topics reflect legitimate concerns 
about the workings of school systems. 

Analyzing the indicators cited in recent reports 

What evidence has recently convinced the public and the 
media that education in the United States is in sad shape (to the 
extent that the flurry of enthusiasm for reform has been based on 
evidence)? An analysis of the indicators in which the authors of 

reform reports have found cause for gloom and those in which 
they urge improvement could provide another source of ideas for 
data collection. The report authors, along with the press and 
television, succeeded in making a case to the country. The types 
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of evidence they used night reasonably be expected to make sense 
for future information users, too, if collected on a continuing 
basis* 

This does not mean that every fact or impression cited in 
the rush to condemn and uplift the schools deserves to be 
formalized as port of the Common Core of Data, naturally. One of 
NCES's responsibilities should be to place the more sensational 
findings in context. For example, comparisons of achievement 
in the United States and other nations ought to be viewed 
alongside other measures that help to balance and explain them, 
such as data on the proportions of children in various countries 
who attend college or reach the age of 18 with the qualifications 
for college admission. The point is that if international 
comparisons <and other types of evidence on the health of our 
schools) have persuasive force for the public, then they ought to 
be drawn and presented in a way that is as technically defensible 
as possible. 

Drawing analogies to indicators in other fields 

Although statistics on education are not going to be watched 
as closely as those on the economy, some ways of collecting and 
reporting economic data may provide useful models for NCES. For 
example, the index of leading economic indicators and data on 
housing starts both provide clues to the health of the economy, 
and they are especially useful because they provide current data 
that tend to predict future developments. A concerted effort by 
NCES to develop and publicize analogous data could serve the 
worthwhile purpose of giving observers of the education system 
something to look at besides test scores. 

Data exist on the number and characteristics of college 
freshmen considering careers in teaching, for example. These 
data could be publicized as an annual signal about coming changes 
in the teaching force. It might also be possible to aggregate a 
number of measures of educational achievement and attainment into 
on index like the gross national product. Such an index would 
oversimplify matters, of course, but so do the economic indexes 
in widespread use. 

In summary, measures of educational trends that are 
intuitively easy to understand, that capture changes in several 
important dimensions at once, or that tend to predict future 
changes could meet some information needs of the public and 
policymakers. If they were released with some fanfare and 
accompanied by clear explanations, they might also help educate 
people i»bout what goes into educational quality. Analysis of 
various groups' concerns about education and the data they have 
found persuasive in the past can also help guide the development 
of such measures • 
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Describing Education al Inputs and Processes More Fullv 

Alfred North Whitehead said, "Seek simplicity, and distrust 
it." Large-scale data collection is especially useful when it 
produces descriptive data, and simple descriptions of such things 
as the demographic characteristics of students and teachers are 
very useful for researchers, policymakers, and the public. On 
the other hand, broadly based surveys aimed at describing what 
goes on in schools run a serious risk of producing bad 
information. My experience in schools suggests that resources 
often go unused or are used in unexpected ways. My 
recommendations, therefore, are twofold : 

o Do not underestimate the value of the simplest 
demographic statistics. 

o In the effort to describe more complex educational 

resources and processes, probe beneath the surface for 
detailed data on exactly who does what with t not 
resources. 

The val ue of simple statistics 

NCES publications clready contain the types of simple, 
factual data that are indispensable in compiling a portrait of 
the education system. Data on the characteristics of students, 
teachers, schools, and school systems provide a sense of trends 
in education and important background variables for anaJ sis of 
developments at the national and state levels. Time-sen«s data 
represent a unique resource for research and policy analysis, and 
the federal government is particularly well qualified, by virtue 
of its centrality and visibility, to collect such data. 

The value of demographic and other descriptive data 
increases greatly when the data are collected and summarized in a 
consistent way from year to year. When a measure changes, the 
trends it is intended to capture can be exaggerated or masked. 
Therefore, even when there seems to be good reason to tinker with 
a question or an index, the potential improvement should be 
carefully weighed against the loss of comparability. Stability 
in measures should win out in most cases. 

The urge to improve on existing data can find a useful 
outlet in extending data collection to new areas. Early- 
childhood education and other child-care settings are an example 
of such an area, where new developments are occurring (and 
policymakers may eventually see a need to catch up). 

Finally, when descriptive data have been collected, they 
should be published as rapidly as possible. Because delays in 
publication reduce the usefulness of data, the Department of 
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Education should expedite whatever steps are needed for clearance 
and quality control. 



Measuring reeourcs uee a nd Program characteristic* 

Over the past fifteen years or so, evaluations and other 
studies of educational programs have grown far sore sophisticated 
because of the recognition that a program may exist on paper and 
yet make limited or unpredictable differences in classroom 
practice. This problem of implementation deserves analysis as 
it applies to the work of NCES. Although I argued above for some 
oversimplified indicators that will meet the public's information 
pr«x«rences, I «l»o believe that sometimes a simple measure of 
educational resources is worse than no measure. Some statistics 
give a misleading picture of what goes on in schools, and for 
selected issues the extra care and expense of gathering accurate, 
detailed information will be worthwhile. 

A topic that has received recent attention is that of 
microcomputer purchases and use in schools. Local news features 
often report how many schools "have computers" or describe 
parents' fund-raising efforts to buy one or two microcomputers 
for a school. The administrators seem to have correctly judged 
that they would receive as much public credit for buying one 
machine as fifty. NCES should not fall into this trap, however. 
The 1984 edition of The Condition of Education usefully reports 
on ho- students use microcomputers and how many minutes the 
average student user spends with them each week. With the 
addition of some numbers indicating how many students are users, 
this would be a good summary of the key information about 
microcomputer use. It might be even more useful if it were 
supplemented with some data — even anecdotal data — i-ilustrating 
the types of use summarized. What is covered in courses in 
computer literacy, for example? 

Innovative technology is not the only area in which the 
details of resource use deserve scrutiny. The number of teachers 
reportedly present in a district or school may not translate 
directly into measures of class size or even of the actual 
teaching force. Not only do some teachers move around the 
building teaching lessons in music or art, but some of them never 
instruct students. They are resource teachers, assistant 
principals <in name or in fact), or managers of categorical 
programs. I met several of these nonteaching teachers when I did 
fieldwork on Title I of the Elementary and Secondary Education 
Act. In setting up the interviews, I had asked to see Title I 
teachers; several principals arranged for me to meet with people 
who »eld teaching certificates and were counted on the building 
roster as teachers but never worked with students other than to 
teat them. 
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In addition to gathering more detailed data about the use of 
particular resources in schools, NCES might probe more deeply 
intc descriptions of curriculum. At the secondary level, we need 
not only surveys of the courses that students take but also some 
data (probably from teachers) on the topics that the courses 
cover, the relative emphasis given to these topics, and the texts 
used. Elementary teachers can furnish similar information on 
the content they try to cover and the texts they use. Because 
such data can be especially useful in conjunction with measures 
of student achievement, the National Assessment of Educational 
Progress (NAEP) is one logical vehicle for collecting them. 

At a further level of detail, data on instructional 
processes would be useful if some reasonable level of accuracy is 
possible. Again, teacher surveys could indicate how time is used 
during the school day or the clase period, how many interruptions 
occur and what these interruptions are, what is taught in small 
groups and in whole-class instruction, what disciplinary 
incidents occur during a typical day, and so on. Naturally, 
better questions will produce better data, and teachers should 
take a major role in helping to refine any such surveys. 

The analysis and presentation of detailed data on school 
programs must differ in some ways from the analysis and 
presentation of the demographic and fiscal data that NCES has 
most commonly collected. Nationwide aggregates and averages may 
not mean very much. For example, the data in Indicators of 
Education Status and Trends on the presence of remedial courses 
in college undoubtedly obscure massive differences in the level 
of remediation needed in colleges with different levels of 
selectivity. Because of the axtent of variation in such courses, 
there can be legitimate debate about whether the nationwide 
enrollment figures mean anything. 

Reports on course content and instructional processes should 
contain information about the amount of variation found as well 
as the central tendencies. Breakdowns by type of district (large 
city, small city, suburban, rural) and, if possible, by state 
would also be interesting. Finally, data on the educational 
experiences of particular student groups (for example, girls, 
boys, students with handicaps, students attending high-poverty 
schools, low achievers, and those whose native language is not 
English) could inform the assessment and formulation of federal 
policy for these groups. 

In summary, I would argue for the collection of detailed 
data on schools' reso/rct use, curriculum content, and 
instructional processes. The expense of doing such data 
collection well means thaw it must be restricted to only a few 
topics, but even data on selected details of educational practice 
would be useful. They would supplement the bread and butter of 
data collection on the simpler characteristics of students, 
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teachers, and achool eyateme — which should continue with as much 
consistency over time as possible. 



Meaaurlng end Reporting Outcomea 

Undoubtedly, moat of the attention that citizens and 
policymakera give to education atatietica will continue to center 
on student a' test scores. Perhaps, though, NCES will be able to 
make marginal contributiona to a more complete picture of the 
outcomea of education. It can do the following: 

o Accompanying reports of teat acorea with more 
deacription of what waa teated. 

o Giving publicity to outcomea such aa attainment. 

o Seeking, uaing, and explaining meaaurea of higher- 
order akilla. 



Describing the tests 

Reports of students' scores on tests should be closely 
accompanied by deacriptiona of what the teata covered. The 
displays of NAEP results in The Condition of Education indicate 
the general categoriea of knowledge teated, but putting somewhat 
more detail on the pagea that contain these summary tablea and 
graphs would be even better. Intereated readera ahould alao have 
ready acceaa to compariaona between tho NAEP teats' content and 
the content of the standardized tests that school systems 
commonly use. 



Ideally, the media and state and local school systems would 
then follow the federal lead and deacribe what high and low 
scores mean. In any case, NCES should do so. 

Measurin g other outcomes 

Education professionala know that teat acorea do not 
summarize all the intereating outcomea of achooling. NCES and 
the rest of the Department of Education ahould not only continue 
to meaaure other outcomea but alao emphaaize them whenever 
poaaible. Good meaaurea of dropout ratea — and of the number of 
people who obtain general equivalency degrees—are examples of 
such outcome meaaurea. The extent of voting among recent 
high-school graduatea ia another intereating atatiatic to have. 

Emphasizin g higher-order skills 

This recommendation reflecta my aenae that achoola are 
increaaingly teaching to the teata deviaed by teat publiahers and 
atate governmenta, that theae teata tend to meaaure rather simple 
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•kills such as decoding and arithmetic operations, and that at 
some point there may be a backlash from education professionals 
or even employers who notice that other outcomes are being 
neglected. Defining higher-order skills is hard, and measuring 
them is harder, but the attempt would be worthwhile. 

Because I have very little technical knowledge about 
measurement, my recommendations in this area are fairly general. 
However, as a would-be consumer of data on student outcomes, I 
can advocate that NCES continue and perhaps intensify its efforts 
to place test score* in context. 



Quality Control 

Because all statistics imperfectly represent reality, an 
agency that collects and disseminates statistics can never do too 
much to identify their weaknesses, correct the weaknesses xt can 
correct, and explain the ones it cannot. NCES has a further 
challenge stemming from its reliance on diverse state education 
agencies (SEAs) to collect and report standard data. Quality 
control takes different forms for the collection of data from 
national samples and the aggregation of state-collected data: 

o Surveys of national samples will provide the highest- 
quality data on topics where ter. » have varying 
definitions, such as '•dropouts," and their quality can 
be improved if studies of response bias are built in. 

o NCES should not hesitate to take a strong stand with 
SEAs on quality and consistency in the data they 
provide to the federal level. 

Surveys o f national samples 

National surveys like High School and Beyond perform a 
unique service to research and policymaking in education. 
Although expensive, they provide relatively trustworthy data on 
detailed topics and permit the analysis of a variety of 
educational issues at the level of the student, the school, or 
the community. Where inconsistencies arise in state-level data 
because states define terms in different ways, national surveys 
using standard definitions can help resolve the resulting 
questions. 

National studies present issues of response bias that are 
manageable. For example, we know that students tend to have 
erroneous ideas of their parents' income levels and that 
household surveys provide better data on this topic. More could 
be done to identify other sources of faulty data in national 
surveys so that alternative means of data collection (such as 
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different respondent* or differently worded questions) can be 
tried. 

Aggregati on of state data 

Some reliance on SEAs to provide data to NCES is sensible 
and even unavoidable, given the relative amounts of resources 
available for data collection in education at the state and 
federal levels. However, the problems resulting from poor 
quality control in states or inconsistencies across states are 
not entirely unavoidable. NCES should be willing to take a firm 
stand on quality control. 

The overall objective should be for SEAs to recognize that 
they are participating in a process that is centrally controlled 
in order to meet national information needs in a technically 
defensible way. For example, NCES should check the data received 
from SEAs against last year's data, look for internal consistency 
in each state's data, and examine how terms are defined across 
states. Any questions or problems should be raised with the 
state, and all states should recognize that they may have to 
revise their initial submissions. Other quality-control steps 
will probably be necessary as well. 

In this federal system, Washington defers to state authority 
on many issues. Oddities of data collection and analysis should 
not be among them, howe/er. I believe that, although the states 
rightly guard their prerogative to collect their own data in 
cheir own ways, they are probably more willing than NCES thinks 
to go along with a national data program that maintains high 
standards of professional quality. State officials may grumble 
about the imposition of tighter central standards, but some 
friction is inevitable in any intergovernmental endeavor, 
including til-* present system of more loosely controlled 
reporting. 

The Council of Chief State School Officers can be a major 
ally in a nationwide effort to correct the flaws in current data 
aggregation. Other state-based organizations, such as the 
National Governors' Association, the National Conference of State 
Legislatures, and the National Association of State Boards of 
Education, sight also play a role, since their members also have 
n stake in good national data on education. The current process 
of generating papers, summarizing their recommendations, and 
holding hearings can lay the groundwork for specifying action 
steps that SEAs could take. If SEAs and others are not already 
engaged in developing such steps, they should begin soon. 

In summary, although NCES lacks the political or legal power 
to compel states to do much about their collection and analysis 
of data, it can probably muster the professional authority to 
improve the data it collects from states. Together with the 
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further refinement of national surveys, this stancs on quality 
control should 90 far towards correcting*, ths weaknesses currently 
visible in national education statistics. 



Conclusion: Sett ing PrioritiM 

Internal deliberations at NCES as well as this process of 
public consent will undoubtedly generate many good ideas. 
Deciding where to concentrate the agency's efforts will be the 
next problem. Although NCES staff will soon immerse themselves 
xn numerous detailed design issues, a focus on a few overriding 
principles will be in order. I would suggest that the highest 
priority be placed on strengthening the credibility of the data 
through such means as quality control and consistency in data 
series. Next in importance would be the principle of addressing 
information users' concerns about the education system, as best 
these can be inferred in advance. To the extent possible, NCES 
sight then expand its data collection in areas such as the 
detailed description of instructional resources and processes. 
Whatever principles are chosen as the key ones, the important 
thing is to concentrate on a manageable set of priorities in this 
ambitious redesign effort. 
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